Quality assurance and curricular development of medical faculties using graduate surveys: Challenges – proposal of a core questionnaire – implementation guide

Graduate surveys provide valuable information to further improve and develop an academic study program. The aim of this project report is to point out the relevance of these surveys for medical schools and to offer guidance on planning and implementing such surveys so that comparable evaluations of medical degree programs are possible. The authors of this project report were asked by the MFT working subgroup on Quality Assurance in Education to generate quality recommendations for surveying medical graduates. To accomplish this, the questionnaires used by the medical schools to survey graduates were closely inspected and analyzed. A questionnaire containing core and optional questions was created as part of this project. A comprehensive matrix was also developed listing all of the procedural elements of graduate surveys.

• Postal mail: last known contact from time as a student (student admissions office, student affairs office, etc.), parents' address, if available. • Research via the government registry offices (partially free of charge; § 8 VwVfG: As public institutions, universities may request addresses within the scope of administrative assistance); Advance purging of outdated addresses to improve address quality.
The professional mobility of physicians is relatively high: It is often difficult to contact students after graduation (moves, student email addresses are deactivated, private email addresses fall under data protection, etc.). § 8 Verwaltungsverfahrensgesetz (VwVfG)

Should the alumni association be involved?
Alumni associations often maintain good contact with graduates.
• Mailing campaigns are done through the alumni associations which have current address lists.
• Selection bias: Not all graduates join an alumni association. Generally, it is those who have positive attitudes toward the university.
• Contact is only possible if there has been consent to have address information shared for other purposes than those of the alumni association.

GDPR (General Data Protection Regulation)
Letter for mailing

How should the letter be written and what should it contain?
A letter should contain the most important information and have a motivational effect.
• Letter is written by a university official (president, dean of studies) to underscore the importance of the survey.
• The signature of an academic dean or university professor can through familiarity lead to a stronger identification with the medical school and thus to a higher level of participation.
• Personally addressing the letter with the proper title and surname.
• Contact leads to the highest response rates.
• Email messages with a direct personalized link to the survey page is recommended for online surveys, more so than a TAN or password login).

Type of survey/ method
Which method should be used for the survey?
Surveys can be online or paper-based. A mixed-methods approach is also possible.
•Data from online surveys are more quickly available and contain fewer errors • Responsive design is recommended (=automatic adaptation to the device used) • Increased response rate through the additional use of paper-based questionnaires.
• Mixed design of online and paper-based questionnaires has been proven to serve different preferences and enable additional contacts (experience with graduate surveys in medicine).
• Errors can occur when entering data from paper-based questionnaires. The entries also need to be checked when scanning them.
• Staff and material costs for a paper-based questionnaire with mailing are higher than for an online survey.
• Physicians have less time and opportunity to fill out a questionnaire online during working hours. It is easier to fill out a paper-based questionnaire while on night duty.
• Be sure to use identical letters and questionnaire formats when following a mixedmethods approach to prevent method effects as a result of different modi. Some results can be difficult to interpret without the opportunity to compare data. Comparative data can sharpen the view of the medical school's profile and the strengths and weaknesses of the study program.
• Enter into collaborations/build networks (at the state level or beyond) while using a questionnaire that is as uniform as possible.
• The analyses can be performed separately for each university or for the network (anonymized or with the real names of the participating universities).

Data protection
What needs to be taken into account regarding data protection?
Who may be contacted? How must addresses and the survey data be kept? Who may be granted access to the data?, etc. The requirements for data protections have increased in recent years. This involves not only the inclusion of a data protection officer prior to conducting a survey and documentation of data processing (GDPR), but also the application for ethical approval.
To be noted: • • A report that complies with data protection rules should be published: This creates publicity for graduate surveys and also serves as a sign for those surveyed that the data are actually used.
• The internal reports can be prepared selectively or as an automated full report in the form of tables (if applicable, with graphics).
• Alternative: reports written in prose.
• The reports can be published annually or according to another interval (as needed).
• Preparing reports takes up a lot of time and resources.
• Depending on the target group, reports may need to contain additional information or be presented in shortened versions. •

Reminders
How often should graduates be written to?
Reminders are necessary to increase response rates.
• Generally, three reminders are recommended: The response rate increases with each reminder (experience with graduate surveys in medicine).
• Each reminder increases the cost (postage, staff): Take note of the cost-benefit ratio.  Currently there is no obligation in Germany to obtain ethical approval. However, receiving ethical approval for student and graduate surveys complies with the international standards of Good Research Practice.
• Ethical approval is required if the results are published internationally. Ethical approval is required for some German-language publications if genetic, biometric, or so-called health data are reported (GDPR, Art. 4 (13-15)) • There is no uniform standard to date in Germany for applying for the ethical approval of student/graduate surveys.

GDPR (General Data Protection Regulation)
Questionnaire Which questions should be asked? How extensive can the questionnaire be?
Generally, the blueprint of the questionnaire is the central building block for graduate surveys. There are many recommendations about the aspects to be decided upon.  Ideally, the subjective benefits for the respondents (interesting topic, opinion matters) and the subjective costs (time, disclosure of information) are balanced in a survey . Providing incentives should help to initiate a "socially minded process of sharing." The provision of money as unconditional "compensation" increases willingness to participate. Depending on the situation, the amount will be contingent on the target group and can increase the speed of response. A lottery is not as effective as unconditionally paying out money; ballpoint pens, lapel pins and stamps are also weaker than monetary incentives.
• Money for each participation • Raffling off money or valuable objects among the participants who have filled out a questionnaire • Incentives in the form of university souvenirs (promotional gifts such as ballpoint pens, etc.) • Ethical attitudes toward "purchased" votes finds them questionable; risk of sample bias.
• "Inappropriately lavish" incentives can have the opposite effect and undermine the credibility of the study • Too small an incentive--just like ones that are too large--can upset the balance of the exchange relationship. Collaboration Who carries out studies on graduates? WIth whom can I cooperate?
In Germany there are different regional and national networks that act as "players" in the area of graduate surveys. Advantages of collaborations are the synergetic effects and cost savings that come with a centralized organization (e.g., creating and hosting online questionnaires).
• Data and survey are hosted on external servers making access to raw data sometimes limited.

Public relations
How can knowledge about the survey be improved? How can the survey be appropriately announced internally and externally? How do I reach potential participants in advance?
• Point out the importance of survey results early on during university study (e.g., through announcements by the dean of studies to impart the importance to students).
• Maintain a website on how graduates are studied to increase visibility.
• Include the university's press office to announce the survey in advance.
• Networking and maintaining contacts in advance can increase the response rate because participants feel personally included and valued.
• Human resources must be allotted for maintaining contact.
Panel surveys How is a graduate survey at multiple time points organized?
• Obtain consent for additional surveys with the first questionnaire.
• Include the medical association or other partners.
• Problem of panel turnover may be more prominent in the case of mobile or very stressed professional groups.
• Data protection strategy must be drawn up prior to the first survey time point.
• Financing should be ensured over a longer period of time (up to 10 years).

Human resources
What human resources need to be planned? How should the position requirements be formulated for staff members? How is the workload distributed over the year?
• Organization/Address search = administrative assistant • Mailing campaign = student assistant • Checking TANs = student assistant • Analyses = academic staff member (50%) and student assistant. Time set at half a year: Student assistant approximately 20 h/month, administrative assistant approximately 12 h/month, analysis approximately 5 h/month.
• Workload is not distributed equally over the year.

Representativity
How is the representativity of the data verified? •

Survey management
What is the most logical place for organizing a graduate survey? Study program level, department level, university level, external cooperative partners /service providers?
• University-wide coordination. Advantages: synergetic effects through centralized organization; internal comparability of the responses is possible.
• Internal coordination for the medical school or study program. Advantages: higher motivation as a result of decentralized planning and greater familiarity with the particular conditions (subject knowledge).
• External coordination (service provider); mailing campaign / establishing contact through university. Advantages: professional management of the field work; • In the case of centralized, universitywide organization, there is less opportunity to influence the details and possibly tight resources.
• Medical school or study program level: time consuming and work intensive (see point above for "Human resources").
• In the case of external coordination, there is possibly less influence on the details regarding how the survey is conducted.

Time point / Time period
When should a graduate survey be conducted?
• 1.5 to 2 years after graduation to cover both the retrospective view of university study and the career entry phase (clinical experiences, postgraduate education goals).
• Survey time period (duration of field work): 1-4 months • Impressions of university study and details can lose their clarity over time. On the other hand, the experience of studying can be reflected upon in a joboriented context.